Generating representations of group interactions

ABSTRACT

A transcript of a group interaction is generated from audio source data representing the group interaction. The transcript includes a sequence of lines of text, each line corresponding to an audible utterance in the audio source data. A conversation path is generated from the transcript by labeling each transcript line with an identifier identifying the speaker of the corresponding utterance in the audio source data. A representation of the group interaction is generated by associating the conversation path with a set of voice profiles, each voice profile corresponding to an identified speaker in the conversation path.

RELATED APPLICATIONS

This application is a continuation of and claims priority to and claimsthe benefit of U.S. patent application Ser. No. 12/473,916 titled“REPRESENTING GROUP INTERACTIONS,” which has had a title change to“GENERATING REPRESENTATIONS OF GROUP INTERACTIONS,” which was filed inthe United States Patent and Trademark Office on May 28, 2009, and whichhas a current status of “Allowed,” and this application is related toU.S. patent application Ser. No. 13/439,175 titled “REPRESENTING GROUPINTERACTIONS,” which was filed in the United States Patent and TrademarkOffice on Apr. 4, 2012, both of which are incorporated herein byreference in their entirety.

BACKGROUND

People with hearing challenges frequently make use of speech to textconversion, or speech recognition, software. Their challenge ismagnified when such software is applied to transcribe a groupinteraction such as a meeting or a panel discussion. Conventional speechrecognition software focuses primarily on accuracy of transcription andnot on differentiating incoming voice signals, and is often tuned to thecharacteristics of a particular speaker. Therefore, such softwarestruggles to accurately transcribe the proceedings of a groupinteraction where several individuals interact unpredictably. Inaddition, the capability to identify a speaker for each utterance andcapture this information in a compact transcript format that facilitatesstorage and management is highly desirable for this application.

It is further beneficial if the stored transcript can be used toregenerate the group interaction as audio data with some fidelity to theoriginal. However, the output of existing text to speech conversionsoftware is often monotonous, either because the transcript format doesnot record the emotional content of the speech, or the software cannotmake use of such additional information.

In addition, group interactions often make use of, and generateinformation on, physical aids such as whiteboards. Conventional speechto text conversion software, by relying solely on the audio data,therefore neglects an important source of auxiliary information aboutthe group interaction.

BRIEF SUMMARY

According to a first aspect of the invention, there is provided a methodof generating a representation of a group interaction, the methodcomprising: generating a transcript of the group interaction from audiosource data representing the group interaction, the transcriptcomprising a sequence of lines of text, each line corresponding to anaudible utterance in the audio source data; generating a conversationpath from the transcript by labeling each transcript line with anidentifier identifying the speaker of the corresponding utterance in theaudio source data; and generating the representation of the groupinteraction by associating the conversation path with a plurality ofvoice profiles, each voice profile corresponding to an identifiedspeaker in the conversation path.

According to a further aspect of the invention, there is provided amethod of regenerating audio data from a representation of a groupinteraction, the method comprising: extracting, from the representationof the group interaction: a conversation path comprising one or moretranscript lines, each transcript line being labeled with an identifieridentifying a speaker; and a plurality of voice profiles, each voiceprofile corresponding to an identified speaker in the conversation path;and regenerating audio data for the group interaction from theconversation path and the voice profiles.

According to a further aspect of the invention, there is provided asystem for generating a representation of a group interaction, thesystem comprising: a transcription module adapted to generate atranscript of the group interaction from audio source data representingthe group interaction, the transcript comprising a sequence of lines oftext, each line corresponding to an audible utterance in the audiosource data; and a labeling module adapted to: generate a conversationpath from the transcript by labeling each transcript line with anidentifier identifying the speaker of the corresponding utterance in theaudio source data; and generate the representation of the groupinteraction by associating the conversation path with a plurality ofvoice profiles, each voice profile corresponding to an identifiedspeaker in the conversation path.

According to a further aspect of the invention, there is provided acomputer program product including a computer readable medium havingrecorded thereon a computer program for implementing any one of themethods described above. Other aspects of the invention are alsodisclosed.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more embodiments of the invention will now be described withreference to the drawings, in which:

FIG. 1 is a block diagram of a system for group generatingrepresentations of group interactions according to one embodiment of theinvention;

FIG. 2 is a block diagram of a regeneration system for groupinteractions according to one embodiment of the invention;

FIG. 3 is a schematic block diagram of a general purpose computer uponwhich arrangements described can be practiced; and

FIG. 4 is a flow diagram illustrating a method of generating arepresentation of a group interaction according to one embodiment of theinvention.

DETAILED DESCRIPTION

Where reference is made in any one or more of the accompanying drawingsto steps and/or features, which have the same reference numerals, thosesteps and/or features have for the purposes of this description the samefunction(s) or operation(s), unless the contrary intention appears.

Disclosed herein are arrangements for generating a representation of amulti-person group interaction with some or all of the followingcharacteristics:

-   -   the representation identifies speakers of transcript lines;    -   the representation facilitates editing, in particular        re-labeling of transcript lines;    -   the representation has associated data that is auxiliary to the        interaction, such as documents, images, or other media data;    -   the representation has qualifiers conveying information about        the emotional content in which lines were spoken;    -   the representation allows audio data representing the        interaction to be regenerated.

FIG. 1 is a block diagram of a system 100 for generating representationsof group interactions according to one embodiment of the invention. Thesystem 100 will be described with reference to FIG. 4, which is a flowdiagram illustrating a method 400 of generating a representation of agroup interaction carried out by the system 100 according to oneembodiment of the invention. In the system 100, a transcription module120 generates (step 410) a transcript 130 from audio source data 110representing the group interaction according to conventional speech totext processing (speech recognition) such as that provided by the IBM'sViaVoice™ product. The transcript 130 comprises a sequence of lines oftext representing discrete audible utterances in the audio source data110, and their associated times. The audio source data 110 can beobtained from one or more microphones (not shown) (real-timeprocessing), or may be stored on, and retrieved from, a disk or otherrecording medium (batch processing).

The audio source data 110 is also used, along with the transcript 130,by a labeling module 160 to generate (step 420) a conversation path 170.The conversation path 170 contains the data from the transcript 130,supplemented by an identifier for each line of text identifying thespeaker of the corresponding utterance in the audio source data 110. Thelabeling module 160 labels each line with a speaker identifier bycomparing pitch and timing characteristics of the utterances in theaudio source data 110 with one or more voice profiles, each representingand characterizing a unique speaker, stored in a voice profile database150. If a speaker cannot be identified, the line is marked asunidentified in the conversation path 170. The labeling module 160 canbe configured to generate an error message in such a case, to drawattention to any unrecognized participant in the group interaction. In avariation of the system 100, the transcription module 120 and thelabeling module 160 are combined in a single module that carries outboth transcription and labeling in parallel, either in real time orbatch processing mode.

The voice profiles in the voice profile database 150 may be obtainedfrom analysis of training utterances previously input to the system 100,corresponding to each participant in a group interaction. In a furtherembodiment, the labeling module 160 labels transcript lines with speakeridentifiers without reference to a voice profile database 150, butinstead generates voice profiles from analysis of the audio utterancesin the audio source data 110 corresponding to the different transcriptlines, either in real time or in batch processing mode. Conventionalaudio feature extraction and clustering techniques, may be used for thispurpose, with each generated profile corresponding to the features of acluster of similar-featured utterances. The labeling module 160 storesthe generated voice profiles in the voice profile database 150 for lateruse in labeling transcripts 130.

In an alternative embodiment, the audio source data 110 is obtained froma plurality of microphones (not shown), and therefore comprises multiplesynchronized audio streams, each stream corresponding to one microphone.If the system 110 is to operate in batch processing mode, thispartitioning is preserved in the recorded source data 110. The labelingmodule 160 assumes that each microphone is primarily associated with asingle speaker in a structured group interaction such as a paneldiscussion or court hearing. The labeling module 160 then simply labelseach transcript line with the speaker identifier corresponding to thestream from which the corresponding utterance primarily originated. Theprimary origin of each utterance may be determined from the relativeamplitudes of the utterance as recorded in each synchronized audiostream.

The generated conversation path 170 is optionally edited (step 430,shown as a dashed box in FIG. 4) by a user of the system 100 through anediting module 180, which also has access to the audio source data 110.The main purpose of the editing is to correct errors in the speakeridentification in the conversation path 170 by the labeling module 160.A user interface module 140 manages the interaction between the user andthe editing module 180 and facilitates the editing, for example byoffering a drag-and-drop-style interface. The editing module 180 mayalso allow the user to edit the transcript lines to correcttranscription errors by the transcription module 120. The editinghistory of the conversation path 170 may be stored along with theconversation path itself.

As an example, consider the simple exemplary conversation path in Table1, where the left column comprises the successive transcript lines, thecentre column their respective starting times (hh:mm:ss), and the rightcolumn the corresponding speaker identifiers:

TABLE 1 Example conversation path Hi, how are you? 13:04:32 Speaker1 Iam fine 13:04:48 Speaker2 Where is the agenda? 13:04:55 Speaker3 And howare you? 13:05:01 Speaker3

The user of the editing module 180 realizes that the last line has beenincorrectly labeled with the speaker identifier “speaker3” based on thelogical flow of the conversation, and hence uses the editing module 180to re-label that line with the speaker identifier “speaker2”, yieldingthe edited conversation path of Table 2:

TABLE 2 Example conversation path of Table 1, after editing Hi, how areyou? 13:04:32 Speaker1 I am fine 13:04:48 Speaker2 Where is the agenda?13:04:55 Speaker3 And how are you? 13:05:01 Speaker2

The (possibly edited) conversation path 170 is associated (step 440)with a subset 175 of the voice profiles in the voice profile database150 to form a representation 190 of the group interaction. The subset175 comprises each voice profile corresponding to an identified speakerin the conversation path 170. The representation 190 may be stored forlater retrieval or transmitted to a remote location.

In an optional step 450 of the method 400 (shown as a dashed box in FIG.4), the user associates, using the editing module 180, auxiliary data195 with the conversation path 170. The auxiliary data 195 may be mediadata such as snapshot images of whiteboards or demonstration videos, ordocuments. The auxiliary data 195 is generated or retrieved byparticipants in the group interaction or by the user of the editingmodule 180, and stored in a separate database (not shown) associatedwith the location of the audio source data 110. The user of the editingmodule 180 may associate the auxiliary data 195 with the conversationpath 170 with appropriate timing information, for example by associatinga snapshot image of a whiteboard at a particular time with thetranscript line corresponding to that time. Depending on the format ofthe representation 190, the associated auxiliary data could be embeddedwithin the conversation path 170, or, as illustrated in FIG. 1,associated as a distinct item 195 within the representation 190, andlinked to by links inserted in the conversation path 170.

In an alternative embodiment, the system 100 inserts certain qualifiersto the bare lines of text in the transcript 130, conveying how thecorresponding utterance was spoken during the interaction. Suchqualifiers, estimated and inserted by the transcription module 120 orthe labelling module 160 from the pitch and timing characteristics ofthe utterance corresponding to the transcript line, assist in theeventual reproduction of the group interaction with greater authenticityand interest to the listener. One such qualifier could be the emotionalcontent of the corresponding utterance, e.g. humorous, emphatic, orangry. Another such qualifier could be the national or regional accentwith which the utterance was spoken.

FIG. 2 is a block diagram of a regeneration system 200 for groupinteractions according to one embodiment of the invention. The input tothe system 200 is a representation 230 of a group interaction. Therepresentation 230 corresponds to the representation 190 generated bythe system 100 of FIG. 1, and comprises a conversation path 210(corresponding to 170 in FIG. 1), voice profile data 220 (correspondingto 175 in FIG. 1), and (optionally) auxiliary data 225 (corresponding to195 in FIG. 1). A regeneration module 240 regenerates audio data 250 forthe group interaction from the representation 230 according toconventional text to speech processing methods. The audio data 250 maybe passed to an audio system (not shown) for reproduction into sound, orstored on an audio recording medium (not shown).

The regeneration module 240 is capable of modulating the regeneratedspeech data from the transcript lines depending on any qualifiers in theconversation path 210, e.g. emotion or accent as described above. Suchmodulation lends greater interest and authenticity to the final audioreproduction.

The regeneration module 240 is also capable of reproducing relevantparts of the auxiliary data 225 in synchronism with the regeneratedaudio data using the timing information describing how the auxiliarydata 225 is associated with the conversation path 210. According to thisembodiment, if (for example) a page of a document in the auxiliary data225 is associated with a certain transcript line in the conversationpath 210, the regeneration module 240 reproduces that document page atthe same time as the audio data regenerated from that transcript line.

The modules 120, 140, 160, 180, and 240 of FIGS. 1 and 2 may beimplemented using a computer system 300, such as that shown in FIG. 3wherein the modules 120, 140, 160, 180, and 240 may be implemented assoftware executable within the computer system 300. In particular, themodules 120, 140, 160, 180, and 240 are effected by instructions thatare carried out within the computer system 300. The modules may beclassified in two categories, in the first of which the modules 120,160, 180, and 240 perform the processing and in the second of which themodule 140 manages a user interface between the modules in the firstcategory and the user. The software may be stored in a computer readablemedium, including the storage devices described below, for example. Thesoftware is loaded into the computer system 300 from the computerreadable medium, and then executed by the computer system 300. Acomputer readable medium having such software or computer programrecorded on it is a computer program product. The use of the computerprogram product in the computer system 300 preferably effects anadvantageous system for speech to text transcription for groupinteractions.

As seen in FIG. 3, the computer system 300 is formed by a computermodule 301, input devices such as a keyboard 302 and a mouse pointerdevice 303, and output devices including a printer 315, a display device314 and loudspeakers 317. An external Modulator-Demodulator (Modem)transceiver device 316 may be used by the computer module 301 forcommunicating to and from a communications network 320 via a connection321. The network 320 may be a wide-area network (WAN), such as theInternet or a private WAN. Where the connection 321 is a telephone line,the modem 316 may be a traditional “dial-up” modem. Alternatively, wherethe connection 321 is a high capacity (e.g.: cable) connection, themodem 316 may be a broadband modem. A wireless modem may also be usedfor wireless connection to the network 320.

The computer module 301 typically includes at least one processor unit305, and a memory unit 306 for example formed from semiconductor randomaccess memory (RAM) and read only memory (ROM). The module 301 alsoincludes an number of input/output (I/O) interfaces including anaudio-video interface 307 that couples to the video display 314 andloudspeakers 317, an I/O interface 313 for the keyboard 302 and mouse303 and optionally a joystick (not illustrated), and an interface 308for the external modem 316 and printer 315. In some implementations, themodem 316 may be incorporated within the computer module 301, forexample within the interface 308. The computer module 301 also has alocal network interface 311 which, via a connection 323, permitscoupling of the computer system 300 to a local computer network 322,known as a Local Area Network (LAN). As also illustrated, the localnetwork 322 may also couple to the wide network 320 via a connection324, which would typically include a so-called “firewall” device orsimilar functionality. The interface 311 may be formed by an Ethernetcircuit card, a wireless Bluetooth™ or an IEEE 802.11 wirelessarrangement.

The interfaces 308 and 313 may afford both serial and parallelconnectivity, the former typically being implemented according to theUniversal Serial Bus (USB) standards and having corresponding USBconnectors (not illustrated). Storage devices 309 are provided andtypically include a hard disk drive (HDD) 310. Other devices such as afloppy disk drive and a magnetic tape drive (not illustrated) may alsobe used. An optical disk drive 312 is typically provided to act as anon-volatile source of data. Portable memory devices, such optical disks(e.g.: CD-ROM, DVD), USB-RAM, and floppy disks for example may then beused as appropriate sources of data to the system 300.

The components 305 to 313 of the computer module 301 typicallycommunicate via an interconnected bus 304 and in a manner which resultsin a conventional mode of operation of the computer system 300 known tothose in the relevant art. Examples of computers on which the describedarrangements can be practised include IBM-PC's and compatibles, SunSparcstations, Apple Mac™ or like computer systems evolved therefrom.

Typically, the software modules discussed above are resident on the harddisk drive 310 and read and controlled in execution by the processor305. Intermediate storage of such modules and any data fetched from thenetworks 320 and 322 may be accomplished using the semiconductor memory306, possibly in concert with the hard disk drive 310. In someinstances, the software may be supplied to the user encoded on one ormore CD-ROM and read via the corresponding drive 312, or alternativelymay be read by the user from the networks 320 or 322. Still further, thesoftware can also be loaded into the computer system 300 from othercomputer readable media. Computer readable media refers to any storagemedium that participates in providing instructions and/or data to thecomputer system 300 for execution and/or processing. Examples of suchmedia include floppy disks, magnetic tape, CD-ROM, a hard disk drive, aROM or integrated circuit, a magneto-optical disk, or a computerreadable card such as a PCMCIA card and the like, whether or not suchdevices are internal or external of the computer module 301. Examples ofcomputer readable transmission media that may also participate in theprovision of instructions and/or data include radio or infra-redtransmission channels as well as a network connection to anothercomputer or networked device, and the Internet or Intranets includinge-mail transmissions and information recorded on Websites and the like.

The second category of modules mentioned above may be executed toimplement one or more graphical user interfaces (GUIs) to be rendered orotherwise represented upon the display 314. Through manipulation of thekeyboard 302 and the mouse 303, a user of the computer system 300 andthe application may manipulate the interface to provide controllingcommands and/or input to the applications associated with the GUI(s).

The modules 120, 140, 160, 180, and 240 may alternatively be implementedin dedicated hardware such as one or more integrated circuits performingthe functions or sub functions of modules 120, 140, 160, 180, and 240.Such dedicated hardware may include graphic processors, digital signalprocessors, or one or more microprocessors and associated memories.

The foregoing describes only some embodiments of the present invention,and modifications and/or changes can be made thereto without departingfrom the scope and spirit of the invention, the embodiments beingillustrative and not restrictive.

What is claimed is:
 1. A system for generating a representation of agroup interaction, comprising: a memory that stores a plurality of voiceprofiles; and a processor programmed to: generate a transcript of thegroup interaction from audio source data representing the groupinteraction, the transcript comprising a sequence of lines of text, eachline corresponding to an audible utterance in the audio source data;generate a conversation path from the transcript by labeling eachtranscript line with an identifier identifying a speaker of thecorresponding audible utterance in the audio source data; and generatethe representation of the group interaction by associating theconversation path with the plurality of voice profiles, each voiceprofile corresponding to the identified speaker in the conversationpath.
 2. The system of claim 1, where the processor is furtherprogrammed to edit the conversation path by re-labeling at least onetranscript line with a different identifier.
 3. The system of claim 1,where the processor is further programmed to associate the conversationpath with auxiliary data.
 4. The system of claim 3, where the auxiliarydata is media data.
 5. The system of claim 3, where the auxiliary datais a document.
 6. The system of claim 3, where the processor is furtherprogrammed to regenerate audio data from the generated representation ofthe group interaction.
 7. The system of claim 6, where the processor isfurther programmed to reproduce at least part of the auxiliary data insynchronism with the regenerated audio data.
 8. The system of claim 7,where, in being programmed to regenerate the audio data from thegenerated representation of the group interaction, the processor isprogrammed to regenerate the audio data using timing informationassociating the auxiliary data with at least one transcript line.
 9. Thesystem of claim 1, where the audio source data comprises pluralsynchronized audio streams, and where in being programmed to generatethe conversation path from the transcript by labeling each transcriptline with the identifier identifying the speaker of the correspondingaudible utterance in the audio source data, the processor is programmedto label each transcript line with a speaker identifier corresponding toan audio stream from which the corresponding audible utterance primarilyoriginated.
 10. A system for regenerating audio data from arepresentation of a group interaction, comprising: a memory that storesthe representation of the group interaction; and a processor programmedto: extract, from the representation of the group interaction: aconversation path comprising one or more transcript lines, eachtranscript line being labeled with an identifier identifying a speaker;and a plurality of voice profiles, each voice profile corresponding tothe identified speaker in the conversation path; and regenerate audiodata for the group interaction from the conversation path and the voiceprofiles.
 11. The system of claim 10, where the processor is furtherprogrammed to: extract qualifiers from the conversation path; andmodulate the audio data based upon the extracted qualifiers.
 12. Thesystem of claim 10, where the processor is further programmed to:extract auxiliary data from the representation; and reproduce at leastpart of the auxiliary data in synchronism with the regenerated audiodata.
 13. The system of claim 12, where the auxiliary data is mediadata.
 14. The system of claim 12, where the auxiliary data is adocument.
 15. A computer program product for regenerating audio datafrom a representation of a group interaction, comprising: a computerreadable storage device having computer readable program code embodiedtherewith, where the computer readable program code when executed on acomputer causes the computer to: extract, from the representation of thegroup interaction: a conversation path comprising one or more transcriptlines, each transcript line being labeled with an identifier identifyinga speaker; and a plurality of voice profiles, each voice profilecorresponding to the identified speaker in the conversation path; andregenerate audio data for the group interaction from the conversationpath and the voice profiles.
 16. The computer program product of claim15, where the computer readable program code when executed on thecomputer further causes the computer to: extract qualifiers from theconversation path; and modulate the audio data based upon the extractedqualifiers.
 17. The computer program product of claim 15, where thecomputer readable program code when executed on the computer furthercauses the computer to: extract auxiliary data from the representation;and reproduce at least part of the auxiliary data in synchronism withthe regenerated audio data.
 18. The computer program product of claim17, where the auxiliary data is media data.
 19. The computer programproduct of claim 17, where the auxiliary data is a document.